Refining the Wrapper Approach - Smoothed Error Estimates for Feature Selection

In the wrapper approach for feature selection, a popular criterion used is the leave-one-out estimate of the classification error. While being relatively unbiased, the leave-one-out error estimate is nonetheless known to exhibit a large variance, which can be detrimental especially for small samples. We propose reducing its variance (i.e. smoothing) at two levels. At the first level, we smooth the error count using estimates of posterior probabilities; while at the second level, we smooth the posterior probability estimates themselves using Bayesian estimation with conjugate priors. Furthermore, we propose using the jackknife to reduce the bias inherent in Bayesian estimators. We then show empirically that smoothing the error estimate gives improved performance in feature selection.